23 research outputs found
Self-Attention Transducers for End-to-End Speech Recognition
Recurrent neural network transducers (RNN-T) have been successfully applied
in end-to-end speech recognition. However, the recurrent structure makes it
difficult for parallelization . In this paper, we propose a self-attention
transducer (SA-T) for speech recognition. RNNs are replaced with self-attention
blocks, which are powerful to model long-term dependencies inside sequences and
able to be efficiently parallelized. Furthermore, a path-aware regularization
is proposed to assist SA-T to learn alignments and improve the performance.
Additionally, a chunk-flow mechanism is utilized to achieve online decoding.
All experiments are conducted on a Mandarin Chinese dataset AISHELL-1. The
results demonstrate that our proposed approach achieves a 21.3% relative
reduction in character error rate compared with the baseline RNN-T. In
addition, the SA-T with chunk-flow mechanism can perform online decoding with
only a little degradation of the performance
Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularization
The CTC model has been widely applied to many application scenarios because
of its simple structure, excellent performance, and fast inference speed. There
are many peaks in the probability distribution predicted by the CTC models, and
each peak represents a non-blank token. The recognition latency of CTC models
can be reduced by encouraging the model to predict peaks earlier. Existing
methods to reduce latency require modifying the transition relationship between
tokens in the forward-backward algorithm, and the gradient calculation. Some of
these methods even depend on the forced alignment results provided by other
pretrained models. The above methods are complex to implement. To reduce the
peak latency, we propose a simple and novel method named peak-first
regularization, which utilizes a frame-wise knowledge distillation function to
force the probability distribution of the CTC model to shift left along the
time axis instead of directly modifying the calculation process of CTC loss and
gradients. All the experiments are conducted on a Chinese Mandarin dataset
AISHELL-1. We have verified the effectiveness of the proposed regularization on
both streaming and non-streaming CTC models respectively. The results show that
the proposed method can reduce the average peak latency by about 100 to 200
milliseconds with almost no degradation of recognition accuracy.Comment: Submitted to ICASSP 2023(5 pages, 2 figures